OpenCV Camera Calibration

Here, you'll get a chance to try camera calibration and distortion correction for yourself! There are three main steps to this process:

1. Use cv2.findChessboardCorners() to find corners in chessboard images and aggregate arrays of image points (2D image plane points) and object points (3D world points).
2. Use the OpenCV function cv2.calibrateCamera() to compute the calibration matrices and distortion coefficients.
3. Use cv2.undistort() to undistort a test image.

Intrinsics vs. Extrinsics

Intrinsic parameters are those things that are inherent to the camera itself, namely the focal length and optical center as well as the distortion coefficients. These properties remain the same no matter how the camera is positioned within the world. The extrinsic parameters of the calibration process describe how the camera's reference frame is oriented within the world reference frame. For a detailed treatment of image formation, intrinsics, extrinsics and much more, check out the free Udacity Intro to Computer Vision Course.

Computing the mapping between image points and world points depends on both the intrinsic and extrinsic parameters. In this section we'll focus on the intrinsics.

The Strategy

In general, you need to calibrate using many images of the test pattern in order to get a good sampling of points across the image plane. In the exercise below, you'll perform the corner finding step on a couple dozen images to accomplish this.

You won't see the result of the corner finding step directly in the output of the editor like you did in the last exercise, but if you want to investigate the code and output more carefully yourself, fork this repository, which contains all of the code and data.

Calibrate using OpenCV

Given object points, image points, and the (x, y) shape of the image, you're ready to perform the calibration. Check out this tutorial for more detail on camera calibration with OpenCV. You'll call the cv2.calibrateCamera() function like this:

ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, 
                                                  (img.shape[1], img.shape[0]), 
                                                   None, None)

The important outputs to focus on here are mtx and dist, which contain the intrinsic camera matrix and distortion coefficients, respectively. The intrinsic camera matrix is typically written like this:

In the above matrix, f_x and f_y correspond to the focal length (and they are equal when pixels are square), while c_x and c_y correspond to the camera (optical) center in the (x, y) plane.

There are five distortion coefficients in dist and they are given in this order:

where the k's are the radial distortion coefficients and the p's are tangential distortion coefficients. To correct an image for radial and tangential distortion, you'll use the cv2.undistort() function to apply these equations:

To apply undistortion to an image you can call cv2.undistort() like this:

undist = cv2.undistort(img, mtx, dist, None, mtx)

The exercise below is all setup to run as-is. To better visualize the steps that are taking place please check out the Jupyter Notebook in this repository.

Start Quiz:

import numpy as np
import cv2
import glob
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

nx = 8
ny = 6
# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((ny * nx,3), np.float32)
objp[:,:2] = np.mgrid[0:nx, 0:ny].T.reshape(-1,2)

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d points in real world space
imgpoints = [] # 2d points in image plane.

# Make a list of calibration images
images = glob.glob('./Cal*.jpg')

# Step through the list and search for chessboard corners
for idx, fname in enumerate(images):
    img = mpimg.imread(fname)
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

    # Find the chessboard corners
    ret, corners = cv2.findChessboardCorners(gray, (nx, ny), None)

    # If found, add object points, image points
    if ret == True:
        objpoints.append(objp)
        imgpoints.append(corners)


img = cv2.imread('test_image.jpg')
img_size = (img.shape[1], img.shape[0])

# Do camera calibration given object points and image points
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, img_size, None, None)

# Perform undistortion
undist = cv2.undistort(img, mtx, dist, None, mtx)

# Visualize undistortion
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 9))
f.tight_layout()
ax1.imshow(img)
ax1.set_title('Original Image', fontsize=50)
ax2.imshow(undist)
ax2.set_title('Undistorted Image', fontsize=50)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)

User's Answer:

(Note: The answer done by the user is not guaranteed to be correct)

import numpy as np
import cv2
import glob
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

nx = 8
ny = 6
# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((ny * nx,3), np.float32)
objp[:,:2] = np.mgrid[0:nx, 0:ny].T.reshape(-1,2)

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d points in real world space
imgpoints = [] # 2d points in image plane.

# Make a list of calibration images
images = glob.glob('./Cal*.jpg')

# Step through the list and search for chessboard corners
for idx, fname in enumerate(images):
    img = mpimg.imread(fname)
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

    # Find the chessboard corners
    ret, corners = cv2.findChessboardCorners(gray, (nx, ny), None)

    # If found, add object points, image points
    if ret == True:
        objpoints.append(objp)
        imgpoints.append(corners)


img = cv2.imread('test_image.jpg')
img_size = (img.shape[1], img.shape[0])

# Do camera calibration given object points and image points
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, img_size, None, None)

# Perform undistortion
undist = cv2.undistort(img, mtx, dist, None, mtx)

# Visualize undistortion
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 9))
f.tight_layout()
ax1.imshow(img)
ax1.set_title('Original Image', fontsize=50)
ax2.imshow(undist)
ax2.set_title('Undistorted Image', fontsize=50)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)